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© Input/display integrated information processing device. 



© 1. An input/display integrated information pro- 
cessing device has : image display circuit for dis- 
^ playing document and image information; image in- 
^ put circuit through which information of an image 
If} such as a picture, a photograph, documents drawn 
O or written on a sheet presented by the user can be 
lO input; a visual coincidence structure for integrally 
combining a display screen of said image display 
If) circuit with an input screen of said image input 
circuit so as to establish visual coincidence of said 
O screens as viewed from the user; contact information 
detection circuit for, when a finger of the user, the 
fTi sheet presented by the user, or the like makes 
contact with said contact information detection cir- 



cuit, detecting information of a contact position, said 
contact information detection circuit being disposed 
on the whole of said input screen of said image input 
circuit or said display screen of said image display 
circuit; image extraction circuit for automatically ex- 
tracting only a required portion from image informa- 
tion input through said image input circuit; image 
information storage circuit for storing said image 
information; image synthesize circuit for combining 
said image with an image which is already displayed 
and stored; and image information control circuit for 
controlling input/output operations of said image in- 
formation. 
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Background of the Invention 

1. Field of the Invention 

The invention relates to an input/display device 
which is useful as an information terminal of a 
facsimile apparatus, a copying machine, an elec- 
tronic filing apparatus, a word processor, a com- 
puter, or the like. 

2. Related Art of the Invention 

Conventionally, an information apparatus which 
handles image information, such as a facsimile 
apparatus, a copying machine, an electronic filing 
apparatus, a word processor, or a computer is 
assembled and used as an individual or indepen- 
dent apparatus. In an information terminal of a 
facsimile apparatus, a copying machine, an elec- 
tronic filing apparatus, a word processor, a com- 
puter, or the like, functions of displaying and input- 
ting an image are realized by a display device and 
an input device which are independently disposed, 
or a display screen and an input screen are formed 
at different places (sites). 

As the recent advance of technology, an ap- 
paratus such as a hand-written word processor into 
which a character image written on a display 
screen with an accompanying pen can be input has 
been practically used. 

In such a prior art apparatus, however, when an 
image is to be input at an information terminal of a 
facsimile apparatus, a copying machine, an elec- 
tronic filing apparatus, a word processor, a com- 
puter, or the like, the position where the input 
operation is conducted is fixed, and a captured 
image cannot be processed or combined with an- 
other image unless the captured image is output on 
a display device located at another place (site), or 
on a paper sheet or the like. This produces prob- 
lems in that the operation is cumbersome and 
requires a prolonged period, and that the apparatus 
is difficult to operate for the user who is not skilled 
in operation. 

In a hand-written word processor, the input 
operation is conducted by directly writing or draw- 
ing a character or a picture on a display screen 
with an accompanying pen. As compared with the 
case where the writing or drawing operation is 
conducted on a paper sheet, however, the operabil- 
ity is poor. Moreover, such a word processor can- 
not handle an image which has been printed or 
drawn on a paper sheet. Also in the case where the 
input operation is conducted by using a mouse, the 
operability is poor, and it is difficult to write or draw 
a character or an image. 

When pictures or documents are to be trans- 
mitted by a prior art facsimile apparatus, the user 



must previously edit the pictures or documents so 
that they appear on one paper sheet. In the case 
where a document is to be transmitted together 
with a photograph or sentences of a book or a 

5 magazine, for example, preparatory operations are 
required in which the photograph or sentences are 
copied on a paper sheet and the paper sheet is 
then stuck to the document. Even when a replay 
has simple contents, furthermore, a transmittal in- 

w voice for the replay must be prepared before the 
reply is to be transmitted, thereby requiring extra 
works. In order to read information to be transmit- 
ted, sheets bearing the information are mechani- 
cally fed into the apparatus. Therefore, only sheets 

75 of a size in a predetermined range can be used in 
the transmittance. When a document written on a 
sheet of a small size is to be transmitted, therefore, 
the document must previously copied to a sheet of 
a larger size in the predetermined range, thereby 

20 requiring extra works and causing sheets to be 
wasted. 

The apparatus has a further problem in that 
image information to be transmitted cannot be 
checked in advance and therefore is transmitted as 
25 it is even if a read error exists in the image 
information. 

In a word processor with a facsimile function, 
an image to be transmitted can be checked in 
advance on display means. However, the works of 
30 capturing an image printed or written on a sheet 
and transmitting the captured image still requires 
cumbersome handling works and setting oper- 
ations. 

35 Summary of the Invention 

In view of the above-discussed points, it is an 
object of the invention to provide an input/display 
integrated information processing device in which 

40 an image display and an image input are integrated 
with each other, an image captured through a dis- 
play screen is displayed on the capturing place 
(site), thereby allowing the user to directly conduct 
an operation and instructions on the display screen 

45 and an input screen. 

In order to attain the objects, the input/display 
integrated information processing device of the first 
aspect of the invention comprises: image display 
means for displaying document and image informa- 

50 tion; image input means through which information 
of an image such as a picture, a photograph, 
documents drawn or written on a sheet presented 
by the user can be input; a visual coincidence 
structure for integrally combining a display screen 

55 of the image display means with an input screen of 
the image input means so as to establish visual 
coincidence of the screens as viewed from the 
user; contact information detection means for, 
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when a finger of the user, the sheet presented by 
the user, or the like makes contact with the contact 
information detection means, detecting information 
of a contact position, the contact information detec- 
tion means being disposed on the whole of the 
input screen of the image input means or the 
display screen of the image display means; image 
extraction means for automatically extracting only a 
required portion from image information input 
through the image input means; image information 
storage means for storing the image information; 
image synthesize means for combining the image 
with an image which is already displayed and 
stored; and image information control means for 
controlling input/output operations of the image in- 
formation. 

In the input/display integrated information pro- 
cessing device of the second aspect of the inven- 
tion, the image input means comprises: a line light 
source for illuminating a presented article on the 
input screen; a line optical sensor for reading one- 
dimensional light reflected by the presented article; 
sensor drive means for driving the line light source 
and the line optical sensor; and read control means 
for controlling the sensor drive means, the read 
control means comprising background light elimi- 
nation means for, at each reading of one line, 
detecting image information from a difference be- 
tween an output of the line optical sensor in the 
case where the line light source is on, and an 
output of the line optical sensor in the case where 
the line light source is off, thereby reducing a noise 
due to external light from illumination. 

In the input/display integrated information pro- 
cessing device of the third aspect of the invention, 
the image input means further comprises an optical 
filter which disposed between the line light source 
and the line optical sensor and through which only 
light from the line light source is transmitted. 

In the input/display integrated information pro- 
cessing device of the fourth aspect of the invention, 
the image input means further comprises scan 
control means for, in accordance with an output of 
the line optical sensor, controlling the line light 
source and the line optical sensor so as to scan 
only a portion corresponding to the sheet pre- 
sented by the user. 

In the input/display integrated information pro- 
cessing device of the fifth aspect of the invention, 
the image extraction means comprises: object im- 
age extraction means for extracting only a portion 
corresponding to the sheet presented by the user; 
required portion extraction means for, when the 
user previously draws a closed curve enclosing a 
required portion on the presented sheet, detecting 
the closed curve to extract only the required por- 
tion; and enclosure addition means for, when the 
user previously draws a closed curve enclosing a 



required portion on the presented sheet, recogniz- 
ing a shape of the closed curve, and for automati- 
cally adding a suitable enclosure to an image of 
the required portion. 

5 The input/display integrated information pro- 

cessing device of the sixth aspect of the invention 
further comprises: input/edition judgment means 
for, based on action and operation procedures ex- 
erted by the user, automatically judging whether 

w the user intends to newly input an image or to edit 
a displayed image; image vertical relation manage- 
ment means for managing position information of 
vertical relations between plural images input 
through the image input means; and image conver- 

75 sion means for conducting image conversion such 
as positional movement, expansion, reduction, rota- 
tion and deformation on the input image. 

The input/display integrated information pro- 
cessing device of the seventh aspect of the inven- 

20 tion further comprises image division means for, 
using gray level information of an image input 
through the image input means, dividing the image 
into a character portion, a table/figure portion, and 
a photograph/picture portion, etc. 

25 The input/display integrated information pro- 

cessing device of the eighth aspect of the invention 
further comprises: character code input means for 
inputting a character code; character code storage 
means for storing input character codes; and print 

30 means for printing out a stored image and/or char- 
acter codes, whereby the device is provided with a 
word processor function. 

In the input/display integrated information pro- 
cessing device of the ninth aspect of the invention, 

35 the character code input means comprises char- 
acter recognition means for conducting a character 
recognition process, and a character code is input 
by conducting a character recognition on a char- 
acter portion of an image input through the image 

40 input means. 

The input/display integrated information pro- 
cessing device of the tenth aspect of the invention 
further comprises: tag generating means for gen- 
erating the name of the user, an object, a date and 

45 time when image data were input through the im- 
age input means, and for generating a title of the 
image data, as tag information; tag storage means 
for storaging the generated tag information ;and tag 
management means for managing image data in 

50 accordance with tag information of the image data, 
whereby the device is provided with a filing func- 
tion. 

The input/display integrated information pro- 
cessing device of the eleventh aspect of the inven- 
55 tion further comprises: image data recognition 
means for conducting an image recognition pro- 
cess on portions bearing a drawing, a photograph 
or a picture of image data, to encode the image 
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data; an image data base for storing image data or 
the coded image data; and data base management 
means for conducting management such as reg- 
istration, and retrieval on the data base. 

The input/display integrated information pro- 
cessing device of the twelfth aspect of the inven- 
tion further comprises key word generation means 
for generating a key word for data retrieval, from 
tag information of image data, or codes obtained 
by the image data recognition means. 

The input/display integrated information pro- 
cessing device of the thirteenth aspect of the in- 
vention further comprises: image invisible means 
for resorting each line of image data or enciphering 
image data to make the image data invisible; image 
visible means for restoring image data made invisi- 
ble to an original state; and user identification 
means for matching the user with registered per- 
sons for the image data on the basis of ID informa- 
tion such as a password, a signature, or a finger- 
print. 

The input/display integrated information pro- 
cessing device of the fourteenth aspect of the 
invention further comprises facsimile communica- 
tion means for conducting facsimile transmission 
and reception through a telephone line or the like, 
whereby the device is provided with a facsimile 
function. 

The input/display integrated information pro- 
cessing device of the fifteenth aspect of the inven- 
tion further comprises facsimile page management 
means for conducting a page management on im- 
ages transmitted and received through a facsimile 
communication. 

The input/display integrated information pro- 
cessing device of the sixteenth aspect of the inven- 
tion further comprises: stylized format storage 
means for storing image information of a stylized 
format for facsimile transmission; and stylized for- 
mat read means for reading out the stylized format 
and for displaying it on the image display means. 

The input/display integrated information pro- 
cessing device of the seventeenth aspect of the 
invention further comprises: key word extraction 
means for extracting a key word from a character 
string which has been obtained by conducting a 
character recognition on a received image; re- 
ceived image classification means for, using key 
word information extracted by the key word extrac- 
tion means and position information of portions 
divided by the image division means, determining 
an address, a sender and contents of the received 
image, and for adding the determined information 
as tag information to the received image; received 
image retrieval means for reading tag information 
of received image information stored in the image 
information storage means, and for retrieving a 
desired received image on the basis of a key word 



input by the user; retrieval order storage means for 
storing an order of candidate images obtained by 
the received image retrieval means; browsing im- 
age generation means for generating a partial im- 

5 age of or a reduced image of the candidate im- 
ages, and for displaying the generated image on 
the image display means in accordance with the 
candidate order; and image selection means for 
reading out image information which is selected by 

w the user from the candidate images displayed on 
the image display means, from the image informa- 
tion storage means. 

According to the first aspect of the invention, 
the image display means, and the image input 

75 means are visually coincident with each other. 
When an image is to be input, therefore, the user 
can operate the device while checking a read out 
image. Since the contact information detection 
means, and the image input means are visually 

20 coincident with each other, furthermore, an occur- 
rence and position of a contact of a reading object 
can be detected so that an image of the object is 
immediately automatically input, whereby the op- 
eration can be made simple and easy-to-under- 

25 stood. Furthermore, the position where an image is 
to be input, and the operation method may be 
displayed on a display screen. Therefore, the user 
feels the sense of assurance, and the apparatus 
burden of the process of detecting the position of 

30 an image is reduced so that the process accuracy 
is improved. Since an image from which an unnec- 
essary background is automatically eliminated by 
the image extraction means is input to an appara- 
tus, the burden on the operator is reduced so that 

35 the efficiency is enhanced. 

According to the second aspect of the inven- 
tion, each time when an image of one line is read 
out, the difference between the output of the optical 
line sensor when the line light source is on and that 

40 of the line sensor when the line light source is off is 
detected. Therefore, disturbance due to illumination 
can be eliminated. 

According to the third aspect of the invention, 
the optical filter through which only light from the 

45 line light source is transmitted is used, and there- 
fore a function that disturbance due to other light 
can be eliminated is performed in addition to that 
of the second aspect. 

According to the fourth aspect of the invention, 

50 the device is provided with the scan control means, 
and hence prevented from scanning a portion 
where no reading object exists. Consequently, the 
period of the input process can be shortened so 
that the process efficiency is enhanced. 

55 According to the fifth aspect of the invention, a 

required portion can be extracted only by enclosing 
the portion by a simple closed curve with using the 
required portion extraction means, and an enclo- 
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sure corresponding to the shape of the closed 
curve is automatically added to the extracted im- 
age by the enclosure addition means, thereby en- 
hancing the appearance. 

According to the sixth aspect of the invention, 
the device is provided with the input/edition judg- 
ment means for judging whether an image is to be 
automatically input or to be subjected to an edit 
operation, and the image conversion means for 
processing an input image. Consequently, the op- 
erations of processing and editing the captured 
image do not require much labor and can be 
conducted easily. 

According to the seventh aspect of the inven- 
tion, the device is provided with the image division 
means for dividing an input image into a character 
portion, a table/figure portion, and a photo- 
graph/picture portion. Even when the user wishes 
to partly modify the layout in a later step, therefore, 
the user can easily select the portion to be modi- 
fied. 

According to the eighth aspect of the invention, 
the character code input means can beautify char- 
acters and the print means can print out the beau- 
tified characters and an input image. In addition to 
the effects of the first aspect, therefore, the device 
can attain a further effect that a word processor 
function in which an image can easily be incor- 
porated and which has an excellent operability is 
realized. 

According to the ninth aspect of the invention, 
character portions of an input image can automati- 
cally be converted into character codes and then 
input by the character recognition means. There- 
fore, the burden on the operator is reduced so that 
the efficiency is enhanced. 

According to the tenth aspect of the invention, 
the provision of the tag generating means ,the tag 
storage means for storing tag information, and the 
tag management means for managing image data 
in accordance with the tag information realizes a 
filing function in which data can easily be input and 
which has an excellent operability, in addition to 
the effects of the first aspect. Further by the tag 
storage means and a tag management means for 
instructing an image display controlling means to 
display the tag information on the each image data 
, the input data and the title can be displayed with 
each image data,therby to make operator easily 
look the input data or title. 

According to the eleventh aspect of the inven- 
tion, the device is provided with the tag storage 
means for storing tag information, and the tag 
management means for instructing the image dis- 
play control means to display the tag information 
as a tag attached to each image data. Therefore, 
input information such as the date and time when 
the image data is input, the name of the user, and 



the object, and also the title of the image data can 
be displayed together with the image data so that 
the input information and the title of the image data 
can easily be recognized. 
5 According to the twelfth aspect of the invention, 

the provision of the key word generation means 
removes the necessity for the user to attach tag 
information to data, thereby eliminating the labor of 
the user. 

w According to the thirteenth aspect of the inven- 

tion, the provision of the image invisible means for 
making image data invisible can hide image data 
which must be concealed from another person. 
Furthermore, the provision of the image visible 

w means for restoring image data which was made 
invisible to the original state, and the user iden- 
tification means for matching the user with regis- 
tered persons for the image data on the basis of ID 
information such as a password, a signature, or a 

20 fingerprint can allow only the user to see the image 
data, whereby the image data can be kept secret. 

According to the fourteenth aspect of the in- 
vention, in addition to the effects of the first aspect, 
the provision of the facsimile communication 

25 means for conducting facsimile transmission and 
reception through a telephone line or the like can 
realize a facsimile function in which the edit pro- 
cess of an image to be transmitted is simplified, 
image information to be transmitted can be 

30 checked before transmission, the operation is sim- 
ple, and transmission errors of a reduced number 
occur. 

According to the fifteenth aspect of the inven- 
tion, the facsimile page management means for 

35 conducting page management on input and re- 
ceived images is disposed so that, when plural 
pages are to be transmitted, a page to be edited is 
displayed on the forefront window and a page not 
to be edited is not displayed. Therefore, the dis- 

40 play screen is prevented from being complicated. 
When plural pages are received, each of the re- 
ceived pages can be displayed on the image dis- 
play means to be checked, and the pages can be 
returned to the sender after required information is 

45 input through the image input means into the 
pages, thereby improving the efficiency of the 
works. Furthermore, only a required page can be 
printed out so that the number of wasted sheets is 
reduced. 

50 According to the sixteenth aspect of the inven- 

tion, the provision of the stylized format storage 
means for storing image information of a stylized 
format for transmission, and the stylized format 
read means for reading out the stylized format and 

55 for displaying it on the image display means allows 
a cover bearing the destination and the sender in a 
stylized format to be prepared on the spot, and to 
be transmitted as it is or without printing out it, 
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thereby reducing the labor of the user and prevent- 
ing sheets from being wasted. 

According to the seventeenth aspect of the 
invention, the provision of the received image clas- 
sification means for determining the sender and the 
contents of a received image on the basis of a 
result of a character recognition, and for automati- 
cally adding them as tag information allows the 
user to select a received image only by checking 
the tag information, thereby improving the efficien- 
cy of the works. Since the received image retrieval 
means for matching tag information with a keyword 
input by the user to retrieve a desired received 
image is disposed, the burden on the user of 
searching the desired received image can be re- 
duced so that the efficiency of the works is en- 
hanced. 

Brief Description of the Drawings 

Fig. 1 is a diagram showing the configuration of 
an embodiment of the first aspect of the inven- 
tion; 

Fig. 2 is a diagram showing the configuration of 
image input means of the second aspect of the 
invention; 

Fig. 3 is a diagram showing the configuration of 

an optical sensor array and a scanner light 

source which are used in image input means of 

the third aspect of the invention; 

Fig. 4 is a diagram showing the configuration of 

image extraction means of the fifth aspect of the 

invention; 

Fig. 5 is a diagram illustrating the operation of 
the image extraction means; 
Fig. 6 is a diagram specifically showing the 
configuration of boundary position detection 
means in the image extraction means; 
Fig. 7 is a diagram showing the configuration of 
outline recognition means 431 in the image ex- 
traction means; 

Fig. 8 is a diagram showing the configuration of 
an embodiment of the sixth aspect of the inven- 
tion; 

Fig. 9 is a diagram illustrating the operation of 
the sixth aspect of the invention; 
Fig. 10 is a diagram illustrating the operation of 
input/edition judgment means and image vertical 
relation management means of the sixth aspect 
of the invention; 

Fig. 11 is a diagram showing the configuration 
of an embodiment of the seventh aspect of the 
invention; 

Fig. 12 is a diagram showing the configuration 
of an embodiment of the eighth aspect of the 
invention; 

Fig. 13 is a diagram showing the configuration 
of an embodiment of the ninth aspect of the 



invention; 

Fig. 14 is a diagram showing the configuration 
of an embodiment of the tenth aspect of the 
invention; 

5 Fig. 15 is a diagram illustrating the configuration 
of the eleventh .twelfth and thirteenth aspects of 
the invention; 

Fig. 16 is a diagram illustrating the configuration 
of the fourteenth aspect of the invention; 

w Fig. 17 is a diagram showing the configuration 
of fifteenth aspect of the invention; 
Fig. 18 is a diagram showing the configuration 
of sixteenth aspect of the invention; 
Fig. 19 is a diagram showing the operation of 

15 sixteenth aspect of the invention; 

Fig. 20 is a diagram showing the configuration 
of seventeenth aspect of the invention; 
Fig. 21 is a diagram showing the operation of 
the seventeenth aspect of the invention. 

20 

Preferred embodiments of tine Invention 

Fig. 1 is a diagram showing the configuration of 
an embodiment of the invention. In Fig. 1, 1 des- 

25 ignates image display means, 2 designates image 
input means, 3 designates a visual coincidence 
structure, 4 designates image extraction means, 5 
designates contact information detection means, 6 
designates image information storage means, 7 

30 designates image synthesize means, and 8 des- 
ignates image information control means. 

The image display means 1 is a display panel 
which consists of a liquid crystal display device, an 
EL display device, a cathode ray tube, or the like. 

35 The image input means 2 is an image information 
read panel which consists of a light source, an 
optical sensor, etc. The contact information detec- 
tion means 5 consists of a transparent touch panel 
of the pressure sensitive type or the electrostatic 

40 capacity type, and detects a contact of a finger of 
the user or a reading object, and the position of the 
contact. 

As shown in Fig. 1, the input screen of the 
image input means 2 is located by the visual 

45 coincidence structure 3 so that the input screen is 
opposed to the display screen of the image display 
means 1 and the coordinate systems of the two 
input screens coincide with each other. The image 
input means 2 may be placed before or behind the 

50 image display means 1. 

First, an outline of the operation of the embodi- 
ment in the case where the user conducts an 
image input operation will be described. In the 
illustrated example, it is assumed that the image 

55 input means 2 is placed before the image display 
means 1 . 

When the user first presses a sheet against the 
image input means 2 while directing downward the 
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sheet face bearing a picture or sentences, the 
contact information detection means 5 detects the 
contact position, and supplies information of the 
reading position and a trigger signal for the reading 
operation, to the image information control means 
8. In response to an input from the user, the image 
information control means 8 sends a trigger signal 
for initiating the reading operation to the image 
input means 2. At this time, the image which is 
displayed on the display screen opposed to the 
input portion is allowed to remain displayed as it is. 

The information of the position of the input 
portion may be previously determined. In this case, 
the input position may be displayed on the display 
screen of the image display means 1 to indicate 
the position to the user. This can simplify the 
operations such as the positioning operation. 

The image input means 2 which has received 
the trigger signal reads out image data from the 
pressed sheet, and supplies them to the image 
extraction means 4. The image extraction means 4 
extracts only the portion corresponding to the 
pressed sheet. The extracted image data are 
stored together with the position information, etc. in 
the image information storage means 6. At the 
same time, the image information control means 8 
sends a trigger signal to the image display means 
1 and the image synthesize means 7 so that the 
image data stored in the image information storage 
means 6 are displayed on the display screen. As 
shown in Fig. 1, the image synthesize means 7 
which has received the signal from the image in- 
formation control means 8 generates an image 
which is obtained by combining documents Nos. 1 
and 2 with each other as if the image data are 
incorporated at the input position, and supplies the 
generated image to the image display means 1 to 
indicate it to the user. The outline of the operation 
has been described. Next the components will be 
described specifically. 

The operation of the contact information detec- 
tion means 5 will be described specifically. The 
contact information detection means 5 is incor- 
porated with the device in such a manner that the 
screen of a touch sensor of the means (hereinafter, 
the screen is referred to as "the sensor screen") 
overlaps the input screen of the image input means 
2. The contact information detection means 5 de- 
tects whether or not an article is made contact with 
positions of the sensor screen. The detection sys- 
tem is configured by using either of the optical 
type, the transparent conductive film (resistance 
film) type, the electrostatic capacity type, the pres- 
sure sensor type, etc. In accordance with the 
pressing force exerted when the sheet face which 
the user intends to be read is made contact with 
the sensor screen, the contact information detec- 
tion means 5 supplies to the image information 



control means 8 a contact signal indicative of a 
contact between the sensor screen and an article, 
and position information of the range of the con- 
tacting article. When the contact signal is stable in 

5 time and position, the image information control 
means 8 judges that the user presses a sheet to be 
read against the sensor screen, and, when the 
signal is not stable, judges that the contact is 
erroneously done. If it is judged that a sheet to be 

w read is pressed against the sensor screen, the 
image information control means 8 automatically 
sends a signal for the reading operation to the 
image display means 1 and the image input means 
2. The points of starting and ending the image 

w reading operation may be set to be in the range 
which is centered at the pressed area and larger 
than the pressed area by a predetermined length. 
Alternatively, the points are automatically deter- 
mined by the image input means 2. The alternative 

20 will be described in the later description of the 
image input means 2. 

The contact information detection means 5 
may be configured so that the position of image 
information is detected by subjecting an input from 

25 the whole of the sensor screen to an image rec- 
ognition process. In this case, the contact informa- 
tion detection means 5 is not required to be placed 
before the image display means 1 or the image 
input means 2. 

30 As described above, the position of a sheet 

face can be detected automatically by the contact 
information detection means 5, and, in order to 
automatically capture image information, the user is 
requested only to press a sheet to be read out 

35 against the sensor screen. The user is not re- 
quested to, for example, press keys or buttons, and 
the work of sticking a sheet is replaced with the 
operation of reading out the sheet, resulting in that 
the device is clear and easy for the user to op- 

40 erate. In the above, the operation in the case where 
the image input means 2 is placed before the 
image display means 1 has been described. The 
operation in the case where the image input means 
2 is placed behind the image display means 1 is 

45 the strictly same as that described above. The 
invention is not restricted by the positional relation- 
ship between the image display means 1 and the 
image input means 2. In such case the sheet is put 
on the surface of the display means 1 . 

50 Next the configuration and operation of the 

image input means 2 will be described specifically. 

In the illustrated embodiment, the image input 
means 2 is configured by a scan type optical 
sensor array. Fig. 2 is a diagram showing the 

55 configuration of the image input means 2. The 
reference numeral 21 designates a line optical sen- 
sor, 21a designates SELFOC lenses, 21b desig- 
nates optical sensors, 22 designates a line light 
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source, 23 designates sensor drive means, 24 des- 
ignates read control means, 24a designates back- 
ground light elimination means, and 25 designates 
scan control means. The image input means 2 is 
configured so that the intensity of light which is 
emitted from the line light source consisting of 
many LEDs or the like and reflected by a reading 
object as shown in Fig. 2 is measured by the line 
optical sensor 21, whereby the object is read as 
image data. The line optical sensor 21 is config- 
ured by linearly arranging pairs of the SELFOC 
lenses 21a and the optical sensors 21b. 

As shown in Fig. 2, the focal point of each 
SELFOC lens 21a is set to be positioned at a sheet 
face placed on the surface of the transparent touch 
panel of the contact information detection means 5. 
Hereinafter, the plane constituted by focal positions 
of the SELFOC lenses 21a when the line optical 
sensor 21 conducts the scanning operation is re- 
ferred to as "the input plane". 

In the embodiment, the wavelength of light 
emitted from the scanner light source 22 may be 
restricted to that in the infrared region. In this case, 
light emitted from the scanner light source 22 can- 
not be sensed by the eyes of the user, and there- 
fore the user is prevented from feeling discomfort 
such as "glare" due to the light emission. 

Fig. 3 is a diagram showing the configuration of 
the line optical sensor 21 and the scanner light 
source 22. The reference numeral 21c designates 
an IR-pass filter, and 22a designates IR emitting 
devices. 

The IR emitting devices 22a consist of IR 
LEDs, tungsten halogen lamps, or the like, and 
emit light of a wavelength in the infrared region. 
Light of a wavelength in the infrared region is 
invisible to the eyes of a human being. The light 
emitted from the IR emitting devices 22a is re- 
flected and then passed through the IR-pass filter 
21c to enter the optical sensors 21b. The IR-pass 
filter 21c allows only light of a wavelength in the 
infrared region to pass therethrough. Therefore, the 
optical sensors 21b detects only light which has 
been emitted from the scanner light source and 
reflected by the sheet face. Then, the optical sen- 
sors 21b output an electric signal the level of which 
corresponds to the intensity of reflection light en- 
tering the respective sensors as described above, 
whereby the gray level of the sheet in the input 
plane position is measured. Alternatively, the IR- 
pass filter 21c may be positionally exchanged with 
the SELFOC lenses 21a. Also in the alternative, the 
same effects are attained. 

When the image input operation is not con- 
ducted, the line optical sensor 21 always stays at a 
predetermined position (hereinafter, referred to as 
"the origin point"). 



The sensor drive means 23 moves the line 
optical sensor 21 along the scan direction. The 
speed of this movement (hereinafter, referred to as 
"the scan speed") may be determined by the user, 

5 or previously set so as to establish the equimultiple 
relationship. The driving operation is conducted 
each time when a signal from the scan control 
means 25 is received. 

The read control means 24 controls the opera- 

w tion of reading out an image of one line in accor- 
dance with the signal from the scan control means 
25, and also the lighting of the line light source 22. 
The outputs of the optical sensors 21b are pro- 
cessed by the background light elimination means. 

75 Furthermore, positions of pixels of the read out 
image information in the display screen are cal- 
culated on the basis of the given scan time and the 
scan speed. When the operation of reading out an 
image of one line is completed, a signal is issued 

20 to the scan control means 25. 

In order to detect only the intensity of light 
reflected by the sheet face, each time when one 
line is read out, the background light elimination 
means 24a reads the difference between the out- 

25 puts of the optical sensors 21b in the case where 
the line light source 22 is lit on, and those of the 
optical sensors 21b in the case where the line light 
source 22 is lit off, and outputs the reading result 
to the image extraction means 4. The background 

30 light elimination means 24a allows only light emit- 
ted from the line light source 22 to be detected, 
resulting in that only light reflected by the sheet 
face can be detected. This reduces the influence 
due to external light from illumination lighting the 

35 device, so as to obtain a sharper image. 

The scan control means 25 sends to the sen- 
sor drive means 23 a signal corresponding to the 
distance along which the line optical sensor 21 is 
moved. When the image input operation is to be 

40 started, a signal for moving the sensor from the 
origin point to the starting point for the scanning 
operation is supplied to the sensor drive means 23. 
The point of starting a scan operation may be 
given in the form of a command by the user as 

45 described above. Alternatively, using the fact that 
substantially no light is generally reflected by a 
portion where a sheet is not placed, a definition 
may be given as follows: "The scan starting point 
is set at an instance when a zone in which all the 

50 optical sensors 21b output a signal corresponding 
to a black pixel fails to continue along a given 
distance." The starting point may be previously 
set. 

When the line optical sensor 21 is moved to 
55 the scan starting point, a signal for the image input 
operation is sent to the read control means 24. 
During the image input operation, the signal which 
corresponds to the moving speed preset as de- 
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scribed above is sent to the sensor drive means 

23, and the scan control means 25 monitors the 
signal output from the read control means 24. 
When it is not judged that the sensor 21 reaches 
the scan ending point, a trigger signal for the 
reading operation is sent to the read control means 

24. When it is judged that the sensor 21 reaches 
the scan ending point, a signal for moving the line 
optical sensor 21 from the current position to the 
origin point is sent to the sensor drive means 23, 
and the sensor is returned to the origin point. 

The judgment on whether the sensor reaches 
the ending point or not may be given in the form of 
a command by the user as described above. Alter- 
natively, using the fact that substantially no light is 
generally reflected by a portion where a sheet is 
not placed, a definition may be given as follows: 
"The scan ending point is set at an instance when 
a zone in which all the optical sensors 21b output a 
signal corresponding to a black pixel continues 
along a given distance." The ending point may be 
previously set. 

When the sensor drive means 23 completes 
the scanning operation of the whole face of the 
pressed sheet as described above, the operation of 
reading the image is ended. 

In the embodiment, the image input means 2 is 
configured by a scan type optical sensor array. 
The means may be configured by a surface sensor 
in which optical sensors are arranged in a plane. 
When a TFT liquid crystal device is used as a 
sensor, the image input means may be integrated 
with the image display means 1. Alternatively, the 
image input means 2 may be configured by lenses 
and a CCD as in the case of an image receiving 
portion of a camera. 

In the above, the embodiment in which the 
image input means 2 is placed before the image 
display means 1 has been described. In the case 
where the image input means 2 is placed behind 
the image display means 1, the image display 
means 1 is placed under the contact information 
detection means 5, and the image display means 1 
is controlled as described above so as to change 
the optical transmittance of the portion through 
which an image is input, and the focal point of each 
SELFOC lens 21a is formed on a sheet placed on 
the display screen of the image display means 1. 
According to this configuration, when a sheet bear- 
ing information to be transmitted is pressed against 
the display screen, the operation of reading an 
image can be conducted in the exactly same man- 
ner. 

The specific configuration and operation of the 
image input means 2 have been described. Next, 
the image extraction means 4 will be described. 

Fig. 4 is a diagram showing the configuration of 
the image extraction means 4. The reference nu- 



meral 41 designates object image extraction 
means, 42 designates required portion extraction 
means, and 43 designates enclosure addition 
means. 

5 Fig. 5 is a diagram illustrating the operation of 

the image extraction means 4. The image which 
has been read out by the image input means 2 is 
subjected to an extraction operation in which only 
the portion corresponding to the pressed sheet is 

w extracted by the object image extraction means 41 
as shown in (1) of Fig. 5. When a closed curve 
enclosing sentences or a picture to be read is 
drawn on the sheet, the required portion extraction 
means 42 detects the position of the closed curve, 

15 and extracts only the required portion enclosed by 
the closed curve as shown in (2) of Fig. 5. The 
enclosure addition means 43 recognizes the shape 
of the closed curve, and adds an enclosure cor- 
responding to the shape of the closed curve to the 

20 extracted image, thereby enhancing the appear- 
ance. 

The operations of the components of the image 
extraction means 4 will be described specifically. 
As shown in Fig. 4, the object image extraction 
25 means 41 consists of image smooth means 411, 
boundary position decision means 412, background 
pixel conversion means 413, read out image stor- 
age means 414, and object image decision means 
415. 

30 The image input means 2 supplies values of 

the pixels to the image extraction means 4. From 
the pixel values supplied from image read means, 
the image smooth means 411 obtains an average 
value of pixel values of neighborhood pixels, and 

35 supplies the obtained value to the boundary posi- 
tion decision means 412. The pixel values supplied 
from the image input means 2 are sequentially 
supplied also to the read out image storage means 
414, and stored therein. 

40 The boundary position decision means 412 

conducts a process for each scan line, and, from 
the received pixel values, decides the position of 
the boundary between the sheet face which is 
pressed against the input plane by the user, and 

45 the background. 

The specific operation of the boundary position 
decision means 412 will be described with refer- 
ence to Fig. 6. Fig. 6 is a diagram specifically 
showing the configuration of the boundary position 

50 decision means 412. The reference numeral 61 
designates one-scan line pixel storage means, 62 
designates boundary pixel judgment means, 63 
designates boundary pixel position storage means, 
64 designates position smooth means, 65 des- 

55 ignates final boundary position storage means, and 
66 designates circumscribed rectangle detection 
means. 
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The one-scan line pixel storage means 61 
stores pixel values of one scan line which are 
supplied from the image smooth means 41 1 . When 
pixel values of one scan line are input, the bound- 
ary pixel judgment means 62 then obtains the 
position of the boundary between the background 
and the object image, using the general knowledge 
that "a background has a higher gray level, and a 
peripheral portion of an object image (the vicinity of 
the boundary between the image and the back- 
ground) has a lower gray level". The boundary 
pixel judgment means 62 first reads out pixel val- 
ues from the one-scan line pixel storage means 61 
in the sequence directed from the starting point of 
the scan line toward the ending point, and judges 
the initial pixel position x which satisfies the rule 
indicated by expression 1 below, to be an upper 
boundary position between the background and the 
object image. The position information of the pixel 
position is supplied to the boundary pixel position 
storage means 63. When the pixel value indicates 
the brightness, the relationship is inverted. 

"p(x)> thand p(x + 1)^th" (1) 

where p(x) indicates the pixel value at a pixel 
position x, and th indicates a predetermined ar- 
bitrary value. Next, the boundary pixel judgment 
means 62 searches the boundary position in a 
similar manner in the sequence directed from the 
ending point of the scan line toward the starting 
point, and judges the initial pixel position which 
satisfies expression 1 above, to be a lower bound- 
ary position. The position information of the pixel 
position is supplied to the boundary pixel position 
storage means 63. 

The boundary pixel position storage means 63 
stores position information of the upper and lower 
boundary positions of each scan line which is sent 
from the boundary pixel judgment means 62. 

The position smooth means 64 reads out posi- 
tion information of the boundary between the back- 
ground and the object image which is stored in the 
boundary pixel position storage means 63. Among 
pixel positions which are adjacent to the pixel posi- 
tion in question along the main scan direction of 
the image read means, a middle point of pixels 
which are close to each other in position is ob- 
tained as the final boundary position, and supplied 
to the final boundary position storage means 65. 
The position smoothing reduces the influence of 
variation in the boundary position due to a noise, 
so that a smooth boundary is obtained. Next, the 
position smoothing along the subscan direction is 
conducted in the same manner, and the result is 
supplied to the final boundary position storage 
means 65. 



The final boundary position storage means 65 
stores position information of the final boundary 
positions sent from the position smooth means 64. 
The circumscribed rectangle detection means 

5 66 reads out the coordinate positions from the final 
boundary position storage means 65, and detects 
the maximum and minimum values with respect to 
the main scan direction of the image input means 
2, and those with respect to the subscan direction, 

w as position information of a circumscribed rectan- 
gle of the object image. 

In this way, the boundary position decision 
means 412 obtains the boundary position between 
the object image and the background, and the 

75 position of the circumscribed rectangle which en- 
closes the object image. 

The background pixel conversion means 413 
reads out information of boundary coordinates from 
the final boundary position storage means 65. On 

20 the basis of the position information of the cir- 
cumscribed rectangle detected by the circum- 
scribed rectangle detection means 66, the gray 
levels of pixels corresponding to pixels between 
the sides of the circumscribed rectangle and the 

25 boundary coordinates, and among the pixels stored 
in the read out image storage means 414 are 
converted to a value p which satisfies expression 2, 
thereby eliminating the background in the circum- 
scribed rectangle. 

30 

0 ^ p ^ th (2) 

From the images stored in the read out image 
storage means 414, the object image decision 

35 means 415 reads out the image of the portion 
enclosed by the circumscribed rectangle sent from 
the circumscribed rectangle detection means 66, 
and supplies the image as the object image to the 
image information control means 8, and the re- 

40 quired portion extraction means 42. 

As a result, the object image extraction means 
41 enables an extra background portion to be auto- 
matically eliminated as shown in (1) of Fig. 5, 
thereby extracting the portion corresponding to the 

45 pressed sheet. Therefore, works of eliminating ex- 
tra portions in a later step are not necessary. 

The image output from the object image de- 
cision means is supplied to the image information 
control means 8 which in turn controls the image 

50 information storage means 6 so as to store the 
image as image information of the whole face of 
the pressed sheet, and the object image. 

Next, the operation of the required portion ex- 
traction means 42 will be described with reference 

55 to Fig. 4. As shown in Fig. 4, the required portion 
extraction means 42 consists of extreme point de- 
tection means 421 , closed curve trace means 422, 
closed curve position storage means 423, and in- 
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ternal image extraction means 424. 

The extreme point detection means 421 se- 
quentially scans the object image with starting from 
an end, and sets the first point of a black pixel 
which is larger than a given size and adjacent to 
the white background, as an extreme point of the 
closed curve. The extreme point detection means 
421 outputs position information of the extreme 
point. 

The closed curve trace means 422 scans pix- 
els in the vicinity with starting the extreme point of 
the closed curve in a given rotation direction, and 
detects in sequence boundary points which are 
black pixels initially found and adjacent to the white 
background. The closed curve trace means 422 
outputs position information of the detected bound- 
ary points, as the closed curve position. When the 
scan returns to the starting point, the scan opera- 
tion is ended. The closed curve position storage 
means 423 stores position information supplied 
from the extreme point detection means, and the 
closed curve trace means. 

In order to read out only image information 
inside the closed curve as the internal image, the 
internal image extraction means 424 sends the 
position information stored in the closed curve po- 
sition storage means 423 to the image information 
control means 8. In accordance with the informa- 
tion supplied from the internal image extraction 
means 424, the image information control means 8 
reads out image information of the internal image, 
and sends it to the image display means. In this 
way, only a required portion can be extracted by 
conducting a simple work in which sentences or a 
picture which the user intends to be transmitted is 
enclosed by a curve by means of the required 
portion extraction means 42. In other words, works 
in which a required portion is clipped from the 
sheet with scissors, a cutter or the like, or extra 
portions are erased after a reading operation can 
be eliminated so that the process efficiency is 
enhanced. 

Next, the operation of the enclosure addition 
means 43 will be described specifically with refer- 
ence to Figs. 4 and 7. As shown in Fig. 4, the 
enclosure addition means 43 consists of outline 
recognition means 431, outline image storage 
means 432, and enclosure synthesized image gen- 
eration means 433. 

First, the outline recognition means 431 judges 
the shape of the closed curve, or whether the 
closed curve is circular, or rectangular or has an- 
other shape. Fig. 7 is a diagram showing the con- 
figuration of the outline recognition means 431 . The 
reference numeral 71 designates outline feature 
extraction means, 72 designates an outline dic- 
tionary, 73 designates matching means, and 74 
designates outline decision means. 



The outline feature extraction means 71 se- 
quentially reads out position information from the 
closed curve position storage means 423, with 
starting from the extreme point, and calculates val- 

5 ues which are obtained by normalizing ratios of 
variations of points in the x- and y-axis directions 
with the whole length. The calculated values are 
output as outline feature amounts to a matching 
means 73. The outline dictionary 72 stores outline 

w feature amount information of standard shapes 
such as a circle, an ellipse, and a quadrilateral. The 
matching means 73 portion obtains square errors 
of the feature amounts supplied from the outline 
feature extraction means 71 , and the outline feature 

75 amounts of the standard shapes stored in the out- 
line dictionary, and supplies the values to the out- 
line decision means 74. 

The outline decision means 74 judges the stan- 
dard shape which produces the smallest error, to 

20 be the outline of the closed curve, and further 
checks whether or not a line segment exists in the 
vicinity of each side. Finally, the outline decision 
means 74 decides the enclosure to which the 
closed curve corresponds. The outline image stor- 

25 age means 432 stores image information of the 
enclosures of the closed curve corresponding to 
the standard shapes. 

The enclosure synthesized image generation 
means 433 first reads out image information of the 

30 standard enclosures corresponding to the outlines 
decided by the outline recognition means 431 , from 
the outline image storage means 432, and then 
image information of the image (internal image) 
extracted by the required portion extraction means 

35 42, from the image information storage means 6. 
Finally, the enclosure synthesized image genera- 
tion means 433 combines the read out internal 
image with the enclosure images as shown in (3) of 
Fig. 5, to generate a final enclosed image. The 

40 generated image is again supplied to the image 
information control means 8. As described above, 
the enclosure addition means produces an excel- 
lent effect that the user is required only to conduct 
a simple work in which sentences or a picture to be 

45 transmitted are enclosed by a curve, in order to 
add an enclosure complying with the closed curve, 
thereby enhancing the appearance. 

Fig. 8 is a diagram showing the configuration of 
a second embodiment of the invention. The refer- 

50 ence numeral 83 designates input/edition judgment 
means, 84 designates image conversion means, 
and 85 designates image vertical relation manage- 
ment means. 

The specific operation of the embodiment in 

55 the case where the user can directly touch a dis- 
played image to edit it will be described. 

When the user touches the sensor screen of 
the contact information detection means 5, the in- 
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put/edition judgment means 83 judges whether the 
touch is conducted with the intention of inputting an 
image or with the intention of editing the currently 
displayed image. This judgment is done on the 
basis of information of the contact position and the 
temporal variation of the position. The information 
is supplied from the contact information detection 
means 5. The judgment criterion will be described 
below. 

The judgment that the touch is conducted with 
the intention of inputting an image is done when 
"no currently displayed image exists at the contact 
position" and "the contact position is stable for a 
given period". In contrast, the judgment that the 
touch is conducted with the intention of editing an 
image is done when "a currently displayed image 
exists at the contact position" and "the contact 
position is stable for a given period". When the 
contact position is not stable in time, it is judged 
that the contact is an erroneous one, and nothing is 
done. 

Fig. 9 is a diagram illustrating the operation of 
editing an image which has been input and is 
currently displayed and does not overlap with an- 
other image. First, the user touches the displayed 
object image which is to be edited. At this time, as 
described above, the input/edition judgment means 

83 automatically judges that the image is to be 
edited, and conducts display operations in order to 
notify the user of the image of the current edit 
object. As shown in Fig. 9(1), for example, the 
displayed image is reversed or blinked or the dis- 
play color is changed. The input/edition judgment 
means 83 issues a trigger signal for the editing 
operation to the image conversion means 84. 

In response to the signal from the input/edition 
judgment means 83, the image conversion means 

84 conducts the affine transformation or the like on 
the displayed image to deform the image, and, on 
the basis of the result, changes the image informa- 
tion of the object image stored in the image in- 
formation storage means 6. The image conversion 
process is ended when the user stops touching the 
sensor screen of the contact information detection 
means 5. The user is informed of the end of the 
process by stopping the display operation indica- 
tive of the selection. 

The kind of the deformation such as move- 
ment, or rotation of the image is automatically 
decided in accordance with the contact information 
supplied from the contact information detection 
means 5, as shown in (2) to (4) of Fig. 9. When the 
user touches portion (a) which is in the center area 
of the object image as shown Fig. 9(2) and moves 
the contact portion, for example, the object image 
is moved. When the user touches portion (b) which 
is in the peripheral area of the object image as 
shown Fig. 9(3) and moves the contact portion, the 



object image is reduced or expanded. When the 
user touches portion (c) which is in the vertex area 
of the object image as shown Fig. 9(4) and moves 
the contact portion, the object image is rotated 

5 about the vertex opposing the contacted vertex. 
Although not shown, the selection of either of the 
above-mentioned movement, expansion/reduction, 
and rotation is displayed on the image display 
means 1 by means of reversing the display of the 

w selection portion, displaying a mark or characters, 
or changing the display color, thereby informing 
the user of the selection. This enables the user to 
feel the sense of assurance, and prevents the user 
from erroneously operating the device. 

75 As a result of the above-mentioned image input 

and editing operations, plural images may overlap 
with each other. The image vertical relation man- 
agement means 85 manages the vertical relation 
by adding to an input image a hierarchical label 

20 indicative of the hierarchy. Next, with reference to 
Fig. 10, the operations of the input/edition judgment 
means 83, and the image vertical relation manage- 
ment means 85 which are conducted in an editing 
operation in the case where an image of the edit 

25 object overlaps with another image will be de- 
scribed. 

First, the image vertical relation management 
means 85 adds hierarchical labels to image in- 
formation stored in the image information storage 

30 means 6, in such a manner that a newly input 
image is in the uppermost hierarchy. 

In the case where plural images overlap with 
each other, when the user touches the image in the 
uppermost hierarchy as shown in Fig. 10(1), the 

35 input/edition judgment means 83 sends the trigger 
signal for the editing operation to the image con- 
version means 84 as described above, and the 
image is moved or deformed. In contrast, when the 
user touch an image in a lower hierarchy, the 

40 input/edition judgment means 83 sends to the im- 
age vertical relation management means 85 a sig- 
nal instructing it to transfer the image to the upper- 
most hierarchy. The image vertical relation man- 
agement means 85 which has received the signal 

45 from the input/edition judgment means 83 changes 
the hierarchical labels of the image information 
stored in the image information storage means 6, in 
such a manner that the object image is in the 
uppermost hierarchy. The image display means 1 

50 changes the display so that the object image is 
displayed at the top level as shown in Fig. 10(2). 
The process is then completed. 

In the embodiment, the input/edition judgment 
means 83 judges the input or editing operation on 

55 the basis of the contact information. Alternatively, 
this judgment is conducted on the basis of another 
information. For example, the input and editing 
operations may be switched by pressing a button 
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or the like, or the judgment may be conducted on 
the basis of a voice of the user. 

As described above, according to the invention, 
the operations of processing and editing the cap- 
tured image can be conducted by directly touching 
the display screen, and hence the operation which 
is intuitively understood and does not require cum- 
bersome works can be realized. Furthermore, 
sheets are prevented from being wastefully con- 
sumed. 

Fig. 11 is a diagram showing the configuration 
of a third embodiment of the invention. The refer- 
ence numeral 88 designates image division means, 
881 designates gray-level marginal distribution cal- 
culation means, 882 designates threshold process 
means, 883 designates zone length dispersion cal- 
culation means, 884 designates division point de- 
cision means, 885 designates gray level histogram 
storage means, and 886 designates zone storage 
means. The image division means 88 divides an 
image into portions such as a character portion, a 
picture/photograph portion, and a graph/table por- 
tion, on the basis of properties such as "a char- 
acter string generally elongates along one direc- 
tion", "a picture or a photograph generally has 
plural gray levels", and "a graph or a drawing has 
a large blank area". The operation of the image 
division means 88 will be described. 

The gray-level marginal distribution calculation 
means 881 reads out the position information and 
gray level information of pixels stored in the image 
information storage means 6, and calculates the 
distribution of pixel numbers of each gray level 
which are counted along the horizontal direction 
(hereinafter, the distribution is referred to as "hori- 
zontal gray level histogram"), and the distribution 
of pixel numbers which are counted along the 
vertical direction (hereinafter, the distribution is re- 
ferred to as "vertical gray level histogram"). The 
calculation results are supplied to the gray level 
histogram storage means 885. 

The threshold process means 882 reads out 
the horizontal and vertical gray level histograms 
stored in the gray level histogram storage means 
885, conducts the threshold operation on the histo- 
grams with using a preset value to divide them into 
plural zones, and supplies position information of 
the zones to the zone storage means 886. The 
zone storage means 886 stores the incoming posi- 
tion information of the zones. 

The zone length dispersion calculation means 
883 reads out the position information of the zones 
stored in the zone storage means 886, and cal- 
culates the lengths of the zones. Thereafter, the 
means 883 calculates the dispersion degrees of the 
zone lengths for each combination of continuous 
zones, and supplies the calculation results to the 
division point decision means 884. 



The division point decision means 884 reads 
out the position information of the zones stored in 
the gray level histogram storage means 885, and 
the zone storage means 886. On the basis of the 

5 gray level information, the position information, and 
the dispersion degrees of the zone lengths for each 
combination of continuous zones which are sup- 
plied from the zone length dispersion calculation 
means 883, thereafter, the division point decision 

w means 884 judges "a zone portion which has a 
larger area of a higher gray level (black) and a 
small dispersion degree of the zone lengths, to be 
a character string portion", "a zone portion which 
has a larger area of a lower gray level (white) and a 

75 large dispersion degree of the zone lengths, to be 
a drawing/table portion", and "a zone portion which 
has a larger area of an intermediate gray level 
(gray) and a small dispersion degree of the zone 
lengths, to be a picture/photograph portion". The 

20 boundary points of the portions are set as division 
points. Finally, the image information of the object 
image stored in the image information storage 
means 6 are divided into portions, and attribute 
labels indicative of the kind of the portion (i.e., a 

25 character portion, or a picture/photograph portion) 
are added to the portions, and the contents of the 
image information storage means 6 are changed. 

In this way, the image division means 88 di- 
vides an input image into portions with respect to 

30 the attributes. When the user wishes to input and 
edit an image, therefore, the user is allowed to 
separately edit characters and a picture even if the 
characters and the picture mixedly exist. This 
eliminates the necessity of previously inputting the 

35 characters and the picture in a separate manner, 
thereby reducing the burden of the user. Alter- 
natively, the user may draw a line by a finger on 
the sensor screen of the contact information detec- 
tion means 5 to directly input position information 

40 of the division point into the division point decision 
means 884 so that characters and a picture are 
separated from each other. 

Fig. 12 is a diagram showing the configuration 
of a fourth embodiment of the invention. The refer- 

45 ence numeral 121 designates character code input 
means, 122 designates character code storage 
means, and 123 designates print means. 

The character code input means 121 supplies 
codes of characters and/or symbols which the user 

50 intends to be input, to the character code storage 
means 122 which stores input character codes 
together with the input positions, styles, and pages. 
When the user wishes to print out an input picture 
or characters, a command input by the user is sent 

55 to the image information control means 8. The 
image information control means 8 reads out in- 
formation of character codes of the page which the 
user intends to be printed out, from the character 

14 
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code storage means 122, and supplies the informa- 
tion to the print means 123. Also, the image in- 
formation control means 8 supplies corresponding 
one from the image information stored in the image 
information storage means 6, to the print means 
123. In accordance with incoming information of 
character codes and image information, the print 
means 123 conducts the printing operation, where- 
by the device is provided with a word processor 
function by which the user is allowed to print out 
beautified characters together with image informa- 
tion. Therefore, a picture image can easily be in- 
corporated into sentences, and the editing opera- 
tion can easily be conducted by means of a touch 
operation exerted by a hand. In contrast to the 
operation of incorporating a picture image by using 
a mouse or a light pen, the operation of incorporat- 
ing a picture image can be done by using a sheet 
and a pencil which are usually employed, so that a 
satisfactorily-drawn picture is easily input. When a 
sheet bearing a picture is cut with scissors or the 
like, it is possible to adjust the size of the image to 
be incorporated, thereby providing an operability 
with feeling of cut-and-paste. 

Fig. 13 is a diagram showing the configuration 
of a fifth embodiment of the invention. The refer- 
ence numeral 131 designates character recognition 
means, 132 designates character segmentation 
means, 133 designates feature extraction means, 

134 designates character classification means, and 

135 designates character code output means. The 
character recognition means 131 conducts a char- 
acter recognition process on an image divided as a 
character string portion, to convert it into character 
codes. The operation of the embodiment will be 
described below. 

It is assumed that a hand-written picture or 
sentences on a sheet which the user intends to be 
beautified are previously stored in the image in- 
formation storage means 6 in the manner de- 
scribed above. 

The character segmentation means 132 reads 
out image information which are stored in the im- 
age information storage means 6 and has an at- 
tribute label indicative of a character string portion, 
extracts images of characters, and supplies them 
to the feature extraction means 133. From an in- 
coming character image, the feature extraction 
means 133 calculates amounts indicating the shape 
and size of the character, such as densities of 
components of a contour line in horizontal, vertical 
and oblique directions, and the area of a circum- 
scribed rectangle, as character feature amounts. 
The calculated feature amounts are supplied to the 
character classification means 134. The character 
classification means 134 compares with the incom- 
ing character feature amounts with character fea- 
ture amounts which are previously extracted from 



images of standard characters and correspond to 
the character codes, and decides the character 
code to which the character image provided with 
the incoming character feature amounts corre- 

5 sponds. The corresponding character code is sent 
to the character code output means 135. The char- 
acter code output means 135 supplies incoming 
character codes to the character code storage 
means 122 which in turn sequentially stores the 

w character codes. 

As described above, the operation of the 
means 131 conducts a character recognition on a 
character portion of an input image, and converts 
the portion into character codes, whereby hand- 

75 written characters can be beautified. 

Fig. 14 is a diagram showing the configuration 
of a sixth embodiment of the invention. The refer- 
ence numeral 40 designates tag generation means, 
41 designates tag storage means, and 42 des- 

20 ignates tag management means. The operation of 
the thus configured sixth embodiment of the inven- 
tion will be described. 

The tag generation means 40 generates in- 
formation relating to the image input, such as a 

25 date and time when a picture drawn on a sheet 
face is read, the name of the user, and the object, 
or a title of the image data, as tag information. The 
generated information is stored in the tag storage 
means 41 . 

30 The generation of tag information may be con- 

ducted in the following manner: Tags are displayed 
in the form of a menu so that the user can select a 
required one of the displayed tags. When the user 
touches an applicable portion of the contact in- 

35 formation detection means 5, the menu may be 
changed with another one. Alternatively, a tag may 
be generated by operating a key board. The user 
may write a title of a picture on a sheet with a 
pencil, and the title may be read through the image 

40 input means 2 so that the read data is used as tag 
information. The tag management means 42 in- 
structs the image information control means 8 to 
display tag information of an image data A on the 
image display means 1 in a manner corresponding 

45 to the arrangement state of the image data A 
stored in the image information storage means 6. 
In the embodiment, the tag management means 42 
issues instructions so that the date and time when 
the picture is read are displayed below the picture 

50 as tag information. 

When such tag information is added to image 
data in the manner described above, the user can 
easily know input information and a title of the 
image data. Furthermore, the image data A can be 

55 subjected to a retrieval by checking tag informa- 
tion. 

Finally, the image information control means 8 
controls the image display means 1 so as to dis- 
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play the image data A in accordance with the 
coordinate position and vertical relation between 
the image data A and other image data which are 
stored in the image information storage means 6. 
Furthermore, in accordance with instructions from 
the tag storage means 41 , tag information such as 
the date and time when the image data A are read 
is displayed below the image data A. As described 
above, the provision of the tag generation means, 
the image information storage means, and the tag 
management means can realize an electronic filing 
function in which operations of inputting and out- 
putting data are easy and simple. 

Fig. 15 is a diagram showing the configuration 
of a seventh embodiment of the invention. The 
reference numeral 152 designates image data rec- 
ognition means which conducts an image recogni- 
tion process on a region judged as a drawing, a 
photograph, or a picture by the image division 
means 88, and encodes image data A. The refer- 
ence numeral 153 designates an image data base 
which stores the image data A or coded image 
data, and 154 designates key word generation 
means which automatically generates a key word 
for retrieval of image data, from tag information of 
the image data, or the coded image data A ob- 
tained by the image data recognition means 152. 
The reference numeral 155 designates data base 
management means which conducts data base 
management processes such as registration of the 
image data A or the coded image data A into the 
image data base 153, and retrieval using key words 
obtained from the key word generation means 154. 

The reference numeral 156 designates image 
invisible means which makes the image data A 
invisible by resorting the image data A in the unit 
of one line or enciphering coded image data, and 
157 designates user identification means which 
matches the user with registered persons for image 
data on the basis of ID information such as a 
password, a signature, or a fingerprint. The refer- 
ence numeral 158 designates image visible means 
which, when the user is identified as a registered 
person for image data by the user identification 
means 157, restores the image data A which were 
made invisible by the image invisible means 156 to 
the original state. 

The reference numeral 159 designates page 
information storage means which stores information 
of one screen displayed on the image display 
means 1 (i.e., information indicative of the contents 
of the image data A and the manner of locating the 
image data A on the screen), as page information. 
The reference numeral 160 designates page man- 
agement means which recognizes a page-turn in- 
dication displayed on the image display means 1 
on the basis of the position of the contact informa- 
tion detection means 5 where a finger of the user 



touches, reads out page information corresponding 
to the recognition result from the page information 
storage means 159, and instructs the image in- 
formation control means 8 to display an image 

5 corresponding to the page information on the im- 
age display means 1. The reference numeral 161 
designates index storage means which stores in- 
dexes of page information classified in accordance 
with the contents, and 162 designates index man- 

w agement means which recognizes an index indica- 
tion displayed on the image display means 1 on 
the basis of the position of the contact information 
detection means 5 where a finger of the user 
touches, reads out page information corresponding 

15 to the recognized index from the index storage 
means 161, and instructs the image information 
control means 8 to display an image corresponding 
to the page information on the image display 
means 1 . 

20 The operation of the embodiment in the case 

where abstracts of patent publications are electron- 
ically filed will be described. In the example, the 
image of each abstract is read and displayed on 
the image display means 1, and the data of the 

25 abstract are registered into the image data base 
153. In the registering operation, a character region 
of the abstract is converted into character codes 
and then registered into the image data base 153, 
and images of other drawing or picture regions are 

30 registered as they are. 

In the operation of the embodiment in the 
above-mentioned case, the steps in which the user 
presses the abstract against the contact information 
detection means 5, the image input means 2 reads 

35 the abstract, only the image portion of the abstract 
is extracted by the image extraction means 4, and 
the image data A of the portion is displayed on the 
image display means 1 are conducted in the same 
manner as those of the first embodiment. There- 

40 fore, their detailed description is omitted. 
Hereinafter, the operation in which the data of the 
displayed abstract are registered into the data base 
classified according to technical fields will be de- 
scribed. 

45 In the manner described above, first, the image 

division means 88 divides the image data A into a 
character region, a drawing region, and a pic- 
ture/photograph region, with using properties such 
as "a character string generally elongates along 

50 one direction", "a picture or a photograph generally 
has plural gray levels", and "a graph or a drawing 
has a large blank area". Since an abstract of a 
patent publication is to be filed in the example, the 
image division means 88 divides the image data A 

55 of the abstract into a character region, and a draw- 
ing/picture region, and the data of the regions are 
stored in the image information storage means 6. 
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The image data recognition means 152 reads 
out the drawing/photograph/picture region of the 
image data A stored in the image information stor- 
age means 6, and conducts an image recognition 
process by comparing the feature amounts of a 
drawing of the image data A with feature amounts 
of a standard shape in the same manner as de- 
scribed in conjunction with the character recogni- 
tion means 131, thereby converting the portion into 
codes indicative of a line, a circle, or the like. The 
codes are supplied to the key word generation 
means 154, and the data base management means 
155. In the same manner as described above, the 
character recognition means 131 reads out the 
character region of the image data A stored in the 
image information storage means 6, and conducts 
a character recognition process on the region to 
encode characters in the region. The character 
codes are supplied to the key word generation 
means 154, and the data base management means 
155. In the embodiment, the drawing/picture region 
is encoded so that data of the region can be used 
in a CAD system or the like. Alternatively, the 
image data of the drawing/picture region may be 
supplied as they are to the data base management 
means 155. 

The key word generation means 154 automati- 
cally generates key words for image data retrieval, 
on the basis of the tag information of the image 
data A generated by the tag generation means 40, 
or the character codes output from the character 
recognition means 131 and the codes output from 
the image data recognition means 152. The gen- 
erated key words are supplied to the data base 
management means 155. From the abstract con- 
sisting of the character codes, therefore, a key 
word relating to the patent described in the ab- 
stract is automatically generated. The key word is 
decided based on the occurrence frequencies of 
terms of sentences written in the abstract. 

The abstract which is read through the image 
input means 2 is displayed on the image display 
means 1 . In this display, the character region which 
has been converted into character codes is dis- 
played in the form of an electronic document simi- 
lar to that displayed on a screen of a word proces- 
sor. The drawing in the other region is displayed as 
it is, or in the form of an image. 

The page management means 160 handles the 
display screen as one page, and controls the page 
information storage means 159 so as to store in- 
formation of the one page (i.e., information indica- 
tive of the contents of the sentences and image 
and the manner of locating them on the screen). As 
the number of abstracts which have been read are 
increased, the number of pages is increased. On 
the basis of the page information stored in the 
page information storage means 159, the page 



management means 160 manages the display 
screen of the image display means 1 . Specifically, 
when the user touches with a finger the page-turn 
indication displayed on the image display means 1 , 

5 the page management means 160 recognizes the 
page turn instructed by the user, based on the 
contact position information supplied from the con- 
tact information detection means 5. Depending on 
the recognition result (the pages are to be turned 

w forward, or the pages are to be turned rearward), 
the page management means 160 reads out cor- 
responding page information from the page in- 
formation storage means 159, and instructs the 
image information control means 8 to display a 

75 display screen corresponding to the page informa- 
tion on the image display means 1 . 

The thus configured page management means 
160 enables the user to switch the display screen 
only by touching a specific portion of the sensor 

20 screen of the contact information detection means 
5. Consequently, the user can easily turn the pages 
with a feeling such as that caused in a usual 
operation of turning pages of a book. 

The page information is classified by the user 

25 in accordance with the contents of the display 
screen. The user employs the classification items 
as indexes in a retrieval of a display screen. In the 
embodiment, in order to realize such a retrieval 
function, page information corresponding to the 

30 pages is stored in the index storage means 161. 
On the basis of the page information, the index 
management means 162 manages the display 
screen of the display panel 1. Specifically, when 
the user touches with a finger an index indication 

35 indicative of a classification item and displayed on 
the image display means 1 , the index management 
means 162 recognizes the index indicated by the 
user, based on the contact position information 
supplied from the contact information detection 

40 means 5. Using the technical field of the abstract 
as an index, therefore, a retrieval operation for each 
field can be conducted over the display screen. 

The thus configured index management means 
162 enables a display screen required by the user 

45 to be immediately retrieved and displayed only by 
touching a specific portion of the sensor screen of 
the contact information detection means 5. The 
operation can easily be understood at a glance so 
that the operability is improved. In the above, page 

50 information is classified by the user. Alternatively, 
page information may be classified automatically 
by using key words generated by the key word 
generation means 154. 

The data base management means 155 regis- 

55 ters the image obtained from the image division 
means 88, and the coded sentences and image 
coded by the character recognition means 131 and 
the image data recognition means 152, into the 
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data base 153, together with the key words gen- 
erated by the key word generation means 154. 
Consequently, the character region of the abstract 
in the form of character codes, and the drawing 
region as it is or in the form of an image are 
registered into the data base 153, together with the 
key words for data retrieval. Alternatively, abstracts 
may be registered according to fields, based on 
page information stored in the page information 
storage means 159, and the index storage means 
161. Since key words of abstracts are generated 
by the key word generation means 154, the ab- 
stracts may be classified according to fields, 
whereby the data of the abstracts may be formed 
as a data base system according to fields. 

The image invisible means 156 randomly re- 
sorts (or scrambles) the image data A in the unit of 
one line or enciphers the coded image data A, 
thereby making the image data A invisible. When 
the invisible data are registered into the data base 
153 by the data base management means 155, it is 
possible to prevent the image data A from being 
accessed with using a usual method by another 
person. 

When the invisible data is to be made visible, 
the user inputs ID information such as a password, 
a signature, or a fingerprint. For example, methods 
of inputting ID information include a method 
wherein a number or a signature is written directly 
on the sensor screen of the contact information 
detection means 5, and that wherein information 
which is written on a sheet with a pencil is read 
through the image input means 2. After a password 
or ID information is input, the user identification 
means 157 matches the input information with the 
password or ID information which was registered 
by the data registrant, to judge whether the user is 
the data registrant or not. If the user is judged to 
be the data registrant, the image visible means 158 
restores the invisible image data to the original 
state or be visible on the image display means 1. 
This combination of the image invisible means 156, 
the user identification means 157, and the image 
visible means 158 can hide the image data A which 
must be concealed from another person. The op- 
eration of the seventh embodiment of the invention 
has been described. 

As described above, according to the embodi- 
ment, the image data recognition means 152 en- 
codes the image data A read through the image 
input means 2, whereby effects that data can easily 
be edited or constructed as a data base, and that a 
retrieval can easily be conducted on the data base 
are attained in addition to the effects of the first 
embodiment. As compared with a system in which 
images are stored as they are, the storage capacity 
required for the data base is very small. 



For each of the image data A, furthermore, the 
key word generation means 154 automatically gen- 
erates key words for data retrieval. Therefore, the 
user is not required to take the trouble of inputting 

5 key words. When image data are classified with 
using the key words, the image data can automati- 
cally be constructed as a data base in accordance 
with the classification items, thereby eliminating the 
labor of the user. 

w Fig. 16 is a diagram showing the configuration 

of an eighth embodiment of the invention. The 
reference numeral 1 designates image display 
means, 2 designates image input means, 3 des- 
ignates a visual coincidence structure, 4 designates 

75 image extraction means, 5 designates contact in- 
formation detection means, 6 designates image 
information storage means, 7 designates image 
synthesize means, 8 designates image information 
control means, 123 designates print means, and 50 

20 designates facsimile communication means. The 
operation of the facsimile apparatus of the embodi- 
ment having the above-mentioned configuration will 
be described with reference to the figure. 

First, the operation conducted when a picture 

25 or sentences are read to be transmitted will be 
described. The user first presses a sheet against 
the image input means 2 while directing downward 
the sheet face bearing the picture or sentences. As 
described above, the contact information detection 

30 means 5 supplies information of the reading posi- 
tion and a trigger signal for the reading operation, 
to the image information control means 8. The 
image information control means 8 then sends a 
trigger signal for initiating the reading operation to 

35 the image input means 2. At this time, the image 
which is displayed on the display screen opposed 
to the input portion is allowed to remain displayed 
as it is. 

The information of the position of the input 

40 portion may be previously determined. In this case, 
the input position may be displayed on the display 
screen of the image display means 1 to indicate 
the position to the user. This can simplify the 
operations such as the positioning operation. 

45 The image input means 2 which has received 

the trigger signal reads out image information from 
the pressed sheet, and supplies it to the image 
extraction means 4. The specific configuration and 
operation of the image input means 2 are the same 

50 as those described above. The image extraction 
means 4 extracts only the portion corresponding to 
the pressed sheet, and sends the information of the 
extracted image to the image information control 
means 8. The specific configuration and operation 

55 of the image extraction means 4 are the same as 
those described above. 

The image information control means 8 con- 
trols the image information storage means 6 to 
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store the image information, and reverses the im- 
age so as to be in the normal direction with respect 
to the user. The image information is sent to the 
image display means 1 . The image display means 

1 displays the image information sent from the 
image information control means 8, to exhibit it to 
the user. Consequently, the user can check the 
image to be transmitted by seeing the image dis- 
played on the image display means 1, thereby 
eliminating erroneous transmission due to a read 
error. 

After checking the image to be transmitted, the 
user inputs a command for triggering the transmis- 
sion, and also a facsimile number of the destina- 
tion, into the image information control means 8. In 
response to the trigger signal, the image informa- 
tion control means 8 sends the image information 
currently sent to the image display means 1, and 
also the input facsimile number to the facsimile 
communication means 50. The facsimile commu- 
nication means 50 transmits the incoming image 
information through a line in accordance with the 
input facsimile number, thereby completing the 
transmission process. The operation in transmis- 
sion has been described. 

Next, the operation in reception will be de- 
scribed. When the facsimile communication means 
50 receives image information through the line, the 
means 50 sends the incoming image information to 
the image information control means 8. The image 
information control means 8 adds the facsimile 
number of the sender to the image information, and 
controls the image information storage means 6 to 
store the combination of the image information and 
the number. The image information control means 
8 sends the combination to the print means 123 
which in turn prints the information and the num- 
ber, thereby completing the reception process. 

In the case where a facsimile number is added, 
when the user designates the facsimile number of 
the sender, the image information control means 8 
displays a received image stored in the image 
information storage means 6 and having the same 
facsimile number, on the image display means 1, 
thereby exhibiting the image to the user. 

Furthermore, the case where the received im- 
age displayed on the image display means 1 is 
returned together with a reply written in the image 
will be described. When document No. 2 is to be 
written as a reply in document No. 1 as shown in 
Fig. 16, the user first presses document No. 2 
against a portion where the document is to be 
added. As described above, the image input means 

2 reads document No. 2 in the manner as de- 
scribed above. As shown in Fig. 16, the image 
synthesize means 7 generates an image which is 
obtained by combining documents Nos. 1 and 2 
with each other as if the image data are incor- 



porated at the input position, and supplies the 
generated image to the image display means 1 to 
exhibit it to the user. 

Image information control means 8 adds to the 

5 image information of documents Nos. 1 and 2 
stored in the image information storage means 6, 
the same image label indicating that the image 
information constitutes a single image in which the 
two documents are combined with each other. The 

w image information to which the image label is ad- 
ded is stored in the storage means. When the 
image information control means 8 reads out image 
information from the image information storage 
means 6 on purpose to transmit the image informa- 

75 tion, the control means reads out images to which 
the same image label is added. The image syn- 
thesize means 7 combines the image information 
with each other, and supplies the combined in- 
formation to the image display means 1, and the 

20 facsimile communication means 50. 

Therefore, plural images can be input on the 
spot so as to be combined onto one sheet, thereby 
reducing the burden of the user. A reply can be 
incorporated into a received image and the result- 

25 ing image can be returned to the sender as it is. 
Consequently, it is not required to newly prepare a 
replay sheet, so that sheets are prevented from 
being wastefully consumed and the efficiency of 
the works is enhanced. 

30 Finally, the user inputs a command for reply to 

the image information control means 8 by, for 
example, pressing a key or a button, and then the 
image information control means 8 automatically 
outputs the image information and the facsimile 

35 number added thereto to the facsimile communica- 
tion means 50, whereby the reply process is com- 
pleted. 

Fig. 17 is a diagram showing the configuration 
of a ninth embodiment of the invention. The refer- 

40 ence numeral 60 designates facsimile page man- 
agement means. The operation of the embodiment 
will be described. The facsimile page management 
means 60 manages pages by adding page num- 
bers to image information to be transmitted or 

45 received. When plural sheets of images are to be 
transmitted, for example, the user designates and 
inputs page numbers of the images during the 
process of inputting the images. The facsimile 
page management means 60 adds the designated 

50 page numbers to the image information stored in 
the image information storage means 6. When the 
user inputs the page number of a page which the 
user wishes to see, into the facsimile page man- 
agement means 60, the page is displayed on the 

55 image display means 1. This enables the user to 
check an arbitrary page before transmission so that 
a wrong page is prevented from being transmitted. 
Finally, the user inputs the facsimile number of the 
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destination, and the trigger signal for transmission 
into the image information control means 8, and the 
facsimile communication means 50 then transmits 
the image information, thereby completing the 
transmission process. This configuration allows the 
user to edit plural sheets of images and then 
transmit them. 

Next, the operation in the case of receiving 
plural sheets of images will be described. The 
facsimile page management means 60 adds page 
numbers to pages of received image information 
which is supplied from the facsimile communication 
means 50, and controls the image information stor- 
age means 6 so that the information is stored 
therein. When the user inputs the page number of 
a page which the user wishes to see, into the 
facsimile page management means 60, the image 
information control means 8 reads out image in- 
formation having the page number from the image 
information storage means 6, and the image in- 
formation is displayed on the image display means 
1 so that the user can see the information. In the 
above, the user designates a page number. Alter- 
natively, no page number is designated, and pages 
may automatically be displayed in sequence for a 
fixed period. 

When a page is to be printed, the page num- 
ber of the page is input, and the image information 
control means 8 reads out image information hav- 
ing the page number from the image information 
storage means 6, and sends it to the print means 
123. Finally, the print means 123 prints out the 
image information. 

This configuration enables only an important 
page to be printed so that sheets are prevented 
from being wastefully consumed. Furthermore, an 
arbitrary one of input images can be output in the 
form of a printout so that the device can be used 
also as a copying machine. Alternatively, the user 
may previously set the image information control 
means 8 so that, immediately after an image is 
received, the image is directly sent through the 
image information control means 8 to the print 
means 123 to be printed. 

Fig. 18 is a diagram showing the configuration 
of a tenth embodiment of the invention. The refer- 
ence numeral 871 designates stylized format stor- 
age means, and 872 designates stylized format 
read means. The operation of the embodiment will 
be described with reference to Fig. 19. 

The stylized format storage means 871 stores 
image information of stylized formats such as a 
transmittal covering letter bearing the name of the 
user, and the facsimile number as shown in Fig. 
19(1), and also image information of image marks 
respectively indicating the stylized formats. In re- 
sponse to a command input by the user, the styl- 
ized format read means 872 controls the image 



display means 1 so as to display thereon the 
image marks indicating the stylized formats stored 
in the stylized format storage means 871 as shown 
in Fig. 19(2). 

5 When the user selects one of the image marks 

as shown in Fig. 19(3), the stylized format cor- 
responding to the selected image mark is read out 
from the stylized format storage means 871, and 
displayed on the image display means 1. Data to 

w be transmitted are incorporated into the format as 
described above, and then transmitted. This elimi- 
nates the necessity of preparing sheets of stylized 
formats such as a transmittal covering letter, there- 
by reducing the labor of the user and preventing 

15 sheets from being wasted. The selection of the 
image mark may be done by the contact informa- 
tion detection means 5 in the manner described 
above, or by pressing a button or the like. 

Furthermore, the character recognition means 

20 131 conducts a character recognition process on a 
character portion of an input image as described 
above, and converts it into character codes. This 
allows hand-written characters to be beautified and 
thereafter transmitted. The character recognition 

25 means 131 may read the facsimile number of the 
sender written on a received transmittable letter so 
that a transmission process is automatically con- 
ducted without requiring the user to input the fac- 
simile number. The character codes may be trans- 

30 mitted as they are, and at the destination transmit- 
ted data may be received as character codes and 
displayed or printed, whereby images of excellent 
quality can be transmitted to the destination. 

Fig. 20 is a diagram showing the configuration 

35 of an eleventh embodiment of the invention. The 
reference numeral 101 designates key word extrac- 
tion means, 102 designates received image clas- 
sification means, 103 designates received image 
retrieval means, 104 designates retrieval order stor- 

40 age means, 105 designates browsing image gen- 
eration means, and 106 designates image selection 
means. 

The character recognition means 131 conducts 
a recognition process on a received image and 

45 adds character strings to the image information. 
The key word extraction means 101 extracts words 
from the character strings with using a word dic- 
tionary, and a grammar dictionary, and thereafter 
determines the occurrence frequencies of terms, 

50 "characters following Mr., Esq., To whom it may 
concern, or the like", "characters indicating a place 
name, or a name such as a company's name", and 
"characters having a larger size as compared with 
the whole size", as key words of the received 

55 image. 

On the basis of the key words extracted by the 
key word extraction means 101, and position in- 
formation and attitudes of portions divided by the 
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image division means 88, the image classification 
means 102 classifies input images in accordance 
with destination, sender, and kind of contents, with 
using knowledge such as that "characters following 
Mr., Esq., To whom it may concern, or the like 
indicate the name of the destination", "a place 
name, and a name written in an edge portion of a 
sheet indicate the address, and name of a sender", 
and "characters having a larger size indicate the 
kind of contents". Then, the image classification 
means 102 adds key words indicative of the clas- 
sification results to the image information of the 
received images, as tag information, resulting in 
that the received images are automatically clas- 
sified and tag information for classification is added 
to the images. 

Next, the operation of the embodiment in the 
case where the user retrieves a received image 
with using such tag information will be described 
with reference to Fig. 21. When the user inputs a 
key word to be retrieved, the received image re- 
trieval means 103 calculates the degree of co- 
incidence between the input key word and tags 
which are added to the image information stored in 
the image information storage means 6, and out- 
puts information indicative of the coincidence de- 
gree of each image information to the retrieval 
order storage means 104. The retrieval order stor- 
age means 104 stores the information in decreas- 
ing order of coincidence degree. On the basis of 
image information of received candidate images 
stored in the image information storage means 6, 
the browsing image generation means 105 gen- 
erates browsing images each of which is obtained 
by combining tag information with a partial image 
of or a reduced image of the images as shown in 
Fig. 21(1). Next, the browsing image generation 
means 105 sends the generated browsing images 
to the image information control means 8 so that a 
higher candidate image is displayed at a higher 
hierarchy level as shown in Fig. 21(2), and the 
images are displayed on the image display means 
1. 

In order to obtain a desired received image, as 
shown in Fig. 21(3), the user touches one of the 
displayed browsing images to select it. Based on 
the contact position information supplied from the 
contact information detection means 5, the image 
selection means 106 judges that the browsing im- 
age displayed at the position is selected, and 
sends the selection result to the image information 
control means 8 as shown in Fig. 21(3). The image 
information control means 8 reads out the image 
information from the image information storage 
means 6 to display it on the image display means 
1, thereby exhibiting the image information to the 
user. 



As described above, according to the embodi- 
ment, an image which the user wishes to see can 
be read out easily and simply from plural received 
images, so that the burden of the user is reduced. 

5 In the above, the first to eleventh embodiments 

of the invention in which the image input means 2 
is configured by a one-dimensional scanner have 
been described. The invention can be applied in 
the same manner also to a configuration in which 

w the image input means 2 is configured by a two- 
dimensional scanner. The embodiments in which 
the image display means 1, and the image input 
means 2 are configured as separate components 
have been described. The invention can be applied 

75 in the same manner also to a configuration in which 
these two means are integrated with each other in 
a device point of view. Although the embodiments 
in which the image display means 1, the image 
input means 2, and the contact information detec- 

20 tion means 5 are configured as separate compo- 
nents have been described, furthermore, the inven- 
tion can be applied also to a configuration in which 
these three means are integrated with each other in 
a device point of view. 

25 The first to eleventh embodiments of the inven- 

tion in which the image input means 2 is placed 
behind the image display means 1 using a liquid 
crystal have been described. The structure of the 
image display means 1 is not restricted to a liquid 

30 crystal display device, and may be configured by 
any structure such as an EL device, or FED as far 
as the optical transmittance can be changed. In the 
case where the image input means 2 is placed 
before the image display means 1, optical trans- 

35 mission is impertinent to the operation, and there- 
fore the display means may be replaced with dis- 
play means of any other kind such as a cathode 
ray tube, or a plasma display device. 

40 Claims 

1. An input/display integrated information pro- 
cessing device comprising: image display 
means for displaying document and image in- 

45 formation; image input means through which 

information of an image such as a picture, a 
photograph, documents drawn or written on a 
sheet presented by the user can be input; a 
visual coincidence structure for integrally com- 

50 bining a display screen of said image display 

means with an input screen of said image 
input means so as to establish visual coinci- 
dence of said screens as viewed from the 
user; contact information detection means for, 

55 when a finger of the user, the sheet presented 

by the user, or the like makes contact with said 
contact information detection means, detecting 
information of a contact position, said contact 



21 



39 



EP 0 652 505 A1 



40 



information detection means being disposed 
on the whole of said input screen of said 
image input means or said display screen of 
said image display means; image extraction 
means for automatically extracting only a re- 5 
quired portion from image information input 
through said image input means; image in- 
formation storage means for storing said image 
information; image synthesize means for com- 
bining said image with an image which is al- w 
ready displayed and stored; and image in- 
formation control means for controlling in- 
put/output operations of said image informa- 
tion. 

15 

An input/display integrated information pro- 
cessing device according to claim 1, wherein, 
in the integral combination in said visual co- 
incidence structure, said image input means is 
placed before said image display means as 20 
viewed from the user. 

An input/display integrated information pro- 
cessing device according to claim 1, wherein, 
in the integral combination in said visual co- 25 
incidence structure, said image input means is 
placed behind said image display means as 
viewed from the user. 

An input/display integrated information pro- 30 
cessing device according to claim 2 or 3, 
wherein said image input means comprises: a 
line light source for illuminating a presented 
article on said input screen; a line optical sen- 
sor for reading one-dimensional light reflected 35 
by said presented article; sensor drive means 
for driving said line light source and said line 
optical sensor; and read control means for 
controlling said sensor drive means, said read 
control means comprising background light 40 
elimination means for, at each reading of one 
line, detecting image information from a dif- 
ference between an output of said line optical 
sensor in the case where said line light source 
is on, and an output of said line optical sensor 45 
in the case where said line light source is off, 
thereby reducing a noise due to external light 
from illumination. 

An input/display integrated information pro- 50 
cessing device according to claim 4, wherein 
said image input means further comprises an 
optical filter which disposed between said line 
light source and said line optical sensor and 
through which only light from said line light 55 
source is transmitted. 



6. An input/display integrated information pro- 
cessing device according to claim 5, wherein 
said image input means further comprises 
scan control means for, in accordance with an 
output of said line optical sensor, controlling 
said line light source and said line optical 
sensor so as to scan only a portion corre- 
sponding to the sheet presented by the user. 

7. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

6, wherein said image extraction means com- 
prises object image extraction means for ex- 
tracting only a portion corresponding to the 
sheet presented by the user. 

8. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

7, wherein said image extraction means com- 
prises required portion extraction means for, 
when the user previously draws a closed curve 
enclosing a required portion on the presented 
sheet, detecting said closed curve to extract 
only said required portion. 

9. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

8, wherein said image extraction means com- 
prises enclosure addition means for, when the 
user previously draws a closed curve enclos- 
ing a required portion on the presented sheet, 
recognizing a shape of said closed curve, and 
for automatically adding a suitable enclosure to 
an image of said required portion to enhance 
an appearance. 

10. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

9, wherein said device further comprises: in- 
put/edition judgment means for, based on ac- 
tion and operation procedures exerted by the 
user, automatically judging whether the user 
intends to newly input an image or to edit a 
displayed image; image vertical relation man- 
agement means for managing position informa- 
tion of vertical relations between plural images 
input through said image input means; and 
image conversion means for conducting image 
conversion such as positional movement, ex- 
pansion, reduction, rotation and deformation on 
said input image. 

11. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

10, wherein said device further comprises im- 
age division means for, using gray level in- 
formation of an image input through said im- 
age input means, dividing the image into a 
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character portion, a table/figure portion, and a 
photograph/picture portion, etc. 



14. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

13, wherein said device further comprises: a 
tag generating means for generating the name 
of the user, an object, a date and time when 
image data were input through the image input 
means, and for generating a title of the image 
data, as tag information; tag storage means for 
storaging the generated tag information ;and 
tag management means for managing image 
data in accordance with tag information of the 
image data, whereby the device is provided 
with a filing function. 

15. An input/display integrated information pro- 
cessing device according to one of claims 1 to 

14, wherein said device further comprises: im- 
age data recognition means for conducting an 
image recognition process on portions bearing 
a drawing, a photograph or a picture of image 
data, to encode said image data; an image 
data base for storing image data or said coded 
image data; and data base management 
means for conducting management such as 
registration, and retrieval on said data base. 

16. An input/display integrated information pro- 
cessing device according to claim 15, wherein 
said device further comprises key word gen- 
eration means for generating a key word for 
data retrieval, from tag information of image 
data, or codes obtained by said image data 
recognition means. 

17. An input/display integrated information pro- 
cessing device according to one of claims 14 



to 16, wherein said device further comprises: 
image invisible means for resorting each line 
of image data or enciphering image data to 
make said image data invisible; image visible 
means for restoring image data made invisible 
to an original state; and user identification 
means for matching the user with registered 
persons for said image data on the basis of ID 
information such as a password, a signature, or 
a fingerprint. 

18. An input/display integrated information pro- 
cessing device according to one of claims 1 to 
17, wherein said device further comprises fac- 
simile communication means for conducting 
facsimile transmission and reception through a 
telephone line or the like, whereby said device 
is provided with a facsimile function. 

19. An input/display integrated information pro- 
cessing device according to claim 18, wherein 
said device further comprises facsimile page 
management means for conducting a page 
management on images transmitted and re- 

25 ceived through a facsimile communication. 

20. An input/display integrated information pro- 
cessing device according to claim 18 or 19, 
wherein said device further comprises: stylized 

30 format storage means for storing image in- 

formation of a stylized format for facsimile 
transmission; and stylized format read means 
for reading out said stylized format and for 
displaying it on said image display means. 

35 

21. An input/display integrated information pro- 
cessing device according to one of claims 18 
to 20, wherein said device further comprises: 
key word extraction means for extracting a key 

40 word from a character string which has been 

obtained by conducting a character recognition 
on a received image; and received image clas- 
sification means for, using key word informa- 
tion extracted by said key word extraction 

45 means and position information of portions di- 

vided by said image division means, determin- 
ing an address, a sender and contents of said 
received image, and for adding said deter- 
mined information as tag information to said 

50 received image. 

22. An input/display integrated information pro- 
cessing device according to claim 21, wherein 
said device further comprises received image 

55 retrieval means for reading tag information of 

received image information stored in said im- 
age information storage means, and for retriev- 
ing a desired received image on the basis of a 



12. An input/display integrated information pro- 
cessing device according to one of claims 1 to 5 
11, wherein said device further comprises: 
character code input means for inputting a 
character code; character code storage means 
for storing input character codes; and print 
means for printing out a stored image and/or w 
character codes, whereby said device is pro- 
vided with a word processor function. 

13. An input/display integrated information pro- 
cessing device according to claim 12, wherein 75 
said character code input means comprises 
character recognition means for conducting a 
character recognition process, and a character 
code is input by conducting a character rec- 
ognition on a character portion of an image 20 
input through said image input means. 
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key word input by the user. 

23. An input/display integrated information pro- 
cessing device according to claim 22, wherein 
said device further comprises: retrieval order 5 
storage means for storing an order of can- 
didate images obtained by said received im- 
age retrieval means; browsing image genera- 
tion means for generating a partial image of or 
a reduced image of said candidate images, w 
and for displaying said generated image on 
said image display means in accordance with 
said candidate order; and image selection 
means for reading out image information which 
is selected by the user from the candidate w 
images displayed on said image display 
means, from said image information storage 
means. 
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